Skip to content

Conversation

@thgreasi
Copy link
Member

@thgreasi thgreasi commented Sep 25, 2025

Resolves: #1031
Change-type: patch
See: https://balena.fibery.io/Work/Project/1926

@thgreasi thgreasi force-pushed the 1031-fix-too-small-buffer-errors-streaming branch 2 times, most recently from 92c0e01 to b316c6e Compare September 25, 2025 14:18
@thgreasi thgreasi changed the title build,deploy: Use streaming while packing the build context files push,build,deploy: Use streaming while packing the build context files Sep 25, 2025
@thgreasi thgreasi force-pushed the 1031-fix-too-small-buffer-errors-streaming branch 9 times, most recently from 59f647d to a613b71 Compare September 26, 2025 07:57
@thgreasi thgreasi changed the title push,build,deploy: Use streaming while packing the build context files push,build: Use streaming while packing the build context files Sep 26, 2025
@thgreasi thgreasi force-pushed the 1031-fix-too-small-buffer-errors-streaming branch from a613b71 to 88bb752 Compare October 13, 2025 09:35
@thgreasi thgreasi requested a review from a team October 13, 2025 16:26
for (const fileStats of filteredFileList) {
pack.entry(
{
void (async () => {
Copy link
Member

@myarmolinsky myarmolinsky Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we create an async function and void it here? Why not keep the for loop as it was before?

Copy link
Member Author

@thgreasi thgreasi Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@myarmolinsky the whole point of this PR is to have this function return the stream w/o having to wait for the whole stream to be constructed (which is what the async-await part does). I'm still (like before) using async-await in here

  • b/c I'm using await for the param to the eolConverter (since I wasn't confident how I could correctly change it to be stream based)
  • for convenience, since it made it easier to (a)wait till all files are added, before we call await preFinalizeCallback & pack.finalize()

So the function returns right away the stream reference, and the void allows us to keep the async-await part working in the background and asynchronously add all files in the stream in order and call all required other functions/callbacks.

If we convert eolConverter to be stream based in a separate PR, we can then get rid of the void-async-await, and replace that by adding event listeners on every operation (tbh, I find the async-await code easier to read).

@thgreasi thgreasi requested a review from myarmolinsky October 31, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Node buffer too small for large images

3 participants